Remote enterprise management of high availability systems

ABSTRACT

A method, system, and program for remote enterprise management of high availability systems are provided. Multiple high availability systems are networked in an enterprise and managed overall by a remote enterprise server. Within each high availability system, a cluster management controller monitors a status of a particular component of the high availability system and reacting to adjust the high availability system when the status indicates an error. In addition, with each high availability system, a monitoring controller detects when the cluster management controller reacts to the status of the particular component and detects a condition of a multiple components of the high availability system. The monitoring controller then reports the error and the condition of the components to the remote enterprise server. The remote enterprise server is enabled to manage each high availability system based on the reports received from each of the multiple high availability systems.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is related to the following co-pendingapplications:

(1) U.S. patent application Ser. No. 10/761,163, filed Jan. 20, 2004,titled “Managing Failover of J2EE Compliant Middleware in a HighAvailability System”.

BACKGROUND OF THE INVENTION

1. Technical Field

The present invention relates in general to improved high availabilitycluster management and, in particular to remote cluster management of ahigh availability system. Still more particularly, the present inventionrelates to improved remote monitoring and management of multiple highavailability systems in an enterprise network.

2. Description of the Related Art

For retailers, banks, and other on-line services where load and demandconstantly fluctuate and where handling each customer request is ofutmost importance, high availability (HA) systems have been developed tohandle mission-critical operations. In general, an HA system is a systemdesigned to eliminate or minimize the loss of service due to eitherplanned or unplanned outages among components of a network system. Thekey method of providing an HA system is through redundant hardware andsoftware components grouped into a cluster of servers.

Redundancy is important in an HA system because when a failure occurs inone node of the cluster, the system transfers the processes performed byone node to another. In a two-node HA cluster, for example, one node istypically designate as the primary node and the other node is typicallydesignated as the backup node. In general, the primary node initiallyruns an application when a cluster is started. In addition, in general,a backup node is designated that will run the application if the primarynode fails. The HA cluster system will typically implement a clustermanager process that periodically polls (or checks the heartbeat) of theprimary node to determine if it is still active. If a “heartbeat” is notdetected, then the cluster manager moves the software process to anotherserver in a cluster.

An important characteristic of an HA system is the recovery time. Ingeneral, the recovery time in a HA system is the time taken for a backupnode to take over an application from a failed primary node. Recoverytime is particularly important in a sales based HA system becauseretailers may lose valuable business if a customer is not able tocomplete transactions quickly. A delay of even 30 seconds for therecovery time diminishes a retailer's business transactions.

Another important characteristic of an HA system is to achieve little orno loss of data during failover. In particular, it is important toachieve little or no loss of committed data. For example, it is notadvantageous to lose valuable information about a customer order orcustomer information during failover.

To achieve a short recovery time and little or no loss of data duringfailure, it is important to initially combine hardware and software insuch as manner that an HA system is built. After a HA system isinitiated, however, it is important to monitor and adjust theconfiguration of the HA system to try to improve the efficiency offailovers and correction of other errors.

When configuring hardware and software for HA systems, many developershave developed customized HA software services to control applicationsin a custom environment which often requires new hardware. Thesesolutions are often expensive and do not take advantage of open sourcetechnologies that allow for portability of applications across multipleplatforms. Further, expensive server systems are often selected, inhopes that the power available in the server system will automaticallyincrease the efficiency of failovers.

As an alternative, open source developers continue to expand open sourcetechnology with functions that can be configured when implementing HAsystems. For example, Linux provides an inexpensive, platformindependent operating system. Developers of Linux continue to addfunctions to the operating system that can be implemented in an opensource manner by other developers. Some of these functions, such as“heartbeat” and distributed replicated block device (drbd), areimplemented with the Linux operating system to assist in configuring HAsystems.

While the Linux tools provide a framework for monitoring for failuresand configuring the hardware used in HA systems, there is a need foradditional monitoring and configuration capability. In particular, thereis a need for a method of monitoring for failures, errors, and othernon-ideal conditions in both the hardware and the software of a HAsystem and for monitoring when the open source HA tools detect failuresand errors. Further, there is a need for remotely accumulating themonitored system status and then remotely facilitating reconfigurationof the HA system.

Moreover, typically multiple HA systems are combined in a network toform an enterprise system. Each HA system may service transactionalrequests for a different store within an enterprise, for example. Thereis a need for a method, system, and program for remotely accumulatingthe monitored system status of multiple HA systems within an enterprise,comparing the system status with performance requirements, and trackinghardware and software needs of each HA system within the enterprise.

Further, when implementing an HA system using an open source operatingsystem framework, it would be advantageous to implement an open sourcecompliant middleware layer to handle transaction requests. Inparticular, it would be advantageous to implement a Java™ 2 platform,Enterprise Edition (J2EE) compliant middleware stack that is: (1)controlled by open source based cluster management interfacing with aremote enterprise console; and (2) able to monitor and configuremultiple HA systems in an enterprise network.

SUMMARY OF THE INVENTION

The present invention provides improved high availability clustermanagement and in particular provides for remote cluster management of ahigh availability system implemented in compliance with an open sourceframework. Still more particularly, the present invention relates toimproved remote monitoring and management of multiple high availabilitysystems in an enterprise network.

According to one aspect of the present invention, multiple highavailability systems are networked in an enterprise and managed overallby a remote enterprise server. Within each high availability system, acluster management controller monitors a status of a particularcomponent of the high availability system and reacts to adjust the highavailability system when the status indicates an error. In addition,with each high availability system, a monitoring controller detects whenthe cluster management controller reacts to the status of the particularcomponent and detects a condition of multiple components of the highavailability system. The monitoring controller then reports the errorand the condition of the components to the remote enterprise server. Theremote enterprise server is enabled to manage the high availabilitysystem based on the report.

In particular, the high availability systems implement a J2EE compliantmiddleware stack monitored by open source functions such as a heartbeatmonitor and a service monitoring daemon. The heartbeat monitor detects,in particular, the status of particular servers on which the middlewarestack resides. The service monitoring daemon detects, in particular thestatus of the particular instances of services provided by themiddleware stack.

The remote enterprise server may determine from the report that aconfiguration change should be made and send a configuration request tothe high availability system. The monitoring controller then adjusts theconfiguration of the high availability system to adjust how theheartbeat monitor or service monitoring daemon will detect and react toerrors. Further, other hardware and software components within the highavailability system may be reconfigured by the monitoring controller.

The remote enterprise server preferably stores monitored informationabout each high availability system in a database. In addition, theenterprise server preferably analyzes the monitored information anddetermines which high availability systems are not meeting performancerequirements. The enterprise server may recommend hardware and softwarechanges and configuration changes. In addition, the enterprise servermay display the comparative performance and provide a real-time displayof the high availability systems and when errors are detected at eachsystem.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features believed characteristic of the invention are setforth in the appended claims. The invention itself however, as well as apreferred mode of use, further objects and advantages thereof, will bestbe understood by reference to the following detailed description of anillustrative embodiment when read in conjunction with the accompanyingdrawings, wherein:

FIG. 1 is a block diagram depicting a server system in which the presentmethod, system, and program may be implemented;

FIG. 2 is a block diagram depicting the hardware configuration of a highavailability cluster for efficient transition of middleware duringfailover;

FIG. 3 is a block diagram depicting a cluster manager in accordance withthe method, system, and program of the present invention;

FIG. 4 is a block diagram depicting one embodiment of a softwareconfiguration of a HA cluster before failover in accordance with themethod, system, and program of the present invention;

FIG. 5 is a block diagram depicting one embodiment of a softwareconfiguration of a HA cluster after failover in accordance with themethod, system, and program of the present invention;

FIG. 6 is a block diagram depicting one embodiment of an implementationof an independent software vendor application within a J2EE compliantmiddleware in a HA system;

FIG. 7 is a high level logic flowchart depicting a process and programfor configuring a drbd partition to a J2EE compliant middleware stack ina HA cluster;

FIG. 8 is a high level logic flowchart depicting a process and programfor controlling configuration and failover of a J2EE compliantmiddleware stack in a HA cluster through a heartbeat controller;

FIG. 9 is a high level logic flowchart depicting a process and programfor controlling a mon function for monitoring services provided by aJ2EE compliant middleware stack;

FIG. 10 is a block diagram depicting an enterprise network includingmultiple HA systems running J2EE middleware stacks in accordance withthe method, system, and program of the present invention; and

FIG. 11 is a high level logic flowchart depicting a process and programfor controlling a monitoring controller within a HA cluster manager inaccordance with the method, system, and program of the presentinvention; and

FIG. 12 is a high level logic flowchart depicting a process and programfor remotely controlling a cluster manager of an HA system toreconfigure the HA system; and

FIG. 13 is a high level logic flowchart depicting a process and programfor controlling a remote enterprise console for managing multiple HAsystems in a cluster.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Referring now to the drawings and in particular to FIG. 1, there isdepicted one embodiment of a system through which the present method,system, and program may be implemented. The present invention may beexecuted in a variety of systems, including a variety of computingsystems, server systems, and enterprise systems.

Computer system 100 includes a bus 122 or other communication device forcommunicating information within computer system 100, and multipleprocessors 112 a–112 n coupled to bus 122 for processing information.Bus 122 preferably includes low-latency and higher latency paths thatare connected by bridges and adapters and controlled within computersystem 100 by multiple bus controllers.

Processor 112 a–112 n may be a general-purpose processor such as IBM'sPowerPC™ processor that, during normal operation, processes data underthe control of operating system and application software accessible froma dynamic storage device such as random access memory (RAM) 114 and astatic storage device such as Read Only Memory (ROM) 116. In a preferredembodiment, multiple layers of software contains machine executableinstructions that when executed on processors 112 a–112 n carry out theoperations depicted in the flowcharts of FIGS. 7, 8, 9, 11, 12, 13 andothers described herein. Alternatively, the steps of the presentinvention might be performed by specific hardware components thatcontain hardwired logic for performing the steps, or by any combinationof programmed computer components and custom hardware components.

The present invention may be provided as a computer program product,included on a machine-readable medium having stored thereon the machineexecutable instructions used to program computer system 100 to perform aprocess according to the present invention. The term “machine-readablemedium” as used herein includes any medium that participates inproviding instructions to processors 112 a–112 n or other components ofcomputer system 100 for execution. Such a medium may take many formsincluding, but not limited to, non-volatile media, volatile media, andtransmission media. Common forms of non-volatile media include, forexample, a floppy disk, a flexible disk, a hard disk, magnetic tape orany other magnetic medium, a compact disc ROM (CD-ROM) or any otheroptical medium, punch cards or any other physical medium with patternsof holes, a programmable ROM (PROM), an erasable PROM (EPROM),electrically EPROM (EEPROM), a flash memory, any other memory chip orcartridge, or any other medium from which computer system 100 can readand which is suitable for storing instructions. In the presentembodiment, an example of a non-volatile medium is mass storage device118 which as depicted is an internal component of computer system 100,but will be understood to also be provided by an external device.Volatile media include dynamic memory such as RAM 114. Transmissionmedia include coaxial cables, copper wire or fiber optics, including thewires that comprise bus 122. Transmission media can also take the formof acoustic or light waves, such as those generated during radiofrequency or infrared data communications.

Moreover, the present invention may be downloaded as a computer programproduct, wherein the program instructions may be transferred from aremote computer such as a server 140 to requesting computer system 100by way of data signals embodied in a carrier wave or other propagationmedium via one of network links 134 a–134 n to a communicationsinterface 132 coupled to bus 122. Communications interface 132 providesa two-way data communications coupling to multiple network links 134a–134 n that may be connected, for example, to a local area network(LAN), wide area network (WAN). When implemented as a server system,computer system 100 typically includes multiple communication interfacesaccessible via multiple peripheral component interconnect (PCI) busbridges connected to an input/output controller. In this manner,computer system 100 allows connections to multiple network computers.

In a network environment, computer system 100 communicates with othersystems through network 102. Network 102 may refer to the worldwidecollection of networks and gateways that use a particular protocol, suchas Transmission Control Protocol (TCP) and Internet Protocol (IP), tocommunicate with one another. Network 102 uses electrical,electromagnetic, or optical signals that carry digital data streams. Thesignals through the various networks and the signals on network links134 a–134 n and through communication interface 132, which carry thedigital data to and from computer system 100, are exemplary forms ofcarrier waves transporting the information. Although not depicted,computer system 100 may also include multiple peripheral components thatfacilitate communication.

When computer system 100 is implemented as a server system in a HAcluster, additional network adapters may be included for supportinglocal connections with other server systems. In addition, whenimplemented as a server system in a HA cluster, computer system 100 maybe designed as a commodity hardware server, such as the xSeries™ serversfrom IBM Corp.

Those of ordinary skill in the art will appreciate that the hardwaredepicted in FIG. 1 may vary. Furthermore, those of ordinary skill in theart will appreciate that the depicted example is not meant to implyarchitectural limitations with respect to the present invention.

With reference now to FIG. 2, there is depicted a block diagram of thehardware configuration of a high availability cluster for efficienttransition of middleware during failover. As illustrated, client systems202 and 204 are connected to network 102 for transferring requests forservices. In the embodiment, client systems 202 and 204 request servicesfrom a high availability (HA) system 208 configured for a quick recoverytime with minimal committed data loss during failover.

As illustrated, HA system 208 includes a primary node 210 and asecondary node 220. As will be further described, primary node 210 andsecondary node 220 preferably implement redundant hardware and softwarethat when executed, provide a high availability system. In particular,primary node 210 and secondary node 220 implement redundant middlewarewhich in a preferred embodiment supports J2EE applications, as will befurther described. Middleware is the software that develops, integrates,and manages web applications and systems. As will be further described,the middleware enables integration of communication, processes, data andthe automation of transaction capacity and systems management.

In particular, Java™ 2 platform, Enterprise Edition (J2EE) provides areusable component model for use in building web applications. J2EEdefines a standard application model, a standard platform for hostingapplications, a compatibility requirement and an operation definition ofthe J2EE platform. An advantage of this open source model is thatmultiple developers can implement the J2EE model with additionalcomponents and configurations, yet all J2EE applications will run on aJ2EE based system.

Developers at International Business Machines, Corp. (IBM™), havedeveloped software that implements the J2EE model. This software oftenfills in gaps not specified in the J2EE framework. For example, IBM™, inparticular, has developed a middleware stack of J2EE compliant softwareproducts that when implemented on a cluster of servers, support J2EEapplications. In general, the middleware stack includes a web server, adatabase server, and a universal Internet application server.Specifically, this stack may include products such as the IBM DB2™ UDBEnterprise Edition, the IBM HTTP Server, and the IBMWebSphere™Application Server.

In addition, primary node 210 and secondary node 220 implementmonitoring and configuration controllers that monitor the failures anderrors of J2EE compliant middleware stack and hardware in a HA cluster.As an example of monitoring and configuration controllers, Tivoli™Monitoring controllers may be implemented that fill in the gaps formonitoring software running in the J2EE framework and facilitate theconfiguration of systems running the J2EE framework.

Primary node 210 and secondary node 220 are connected in a simple,reliable manner that enables each node to quickly check the heartbeat ofthe other node. In the embodiment, this connection is enabled by across-over cable 218 connected between network adapters at each node. Inparticular, cross-over cable 218 preferably enables an Ethernetconnection for transferring heartbeat data. Alternatively, heartbeatdata may also be transferred across the public IP connection via network102 in event that cross-over cable 218 fails. It will be understood thatother hardware may be implemented for providing the heartbeatcommunication channel between primary node 210 and secondary node 220and that in addition to a network based connection, a serial connectionmay be implemented.

In particular, when a heartbeat signal is sent between primary node 210and secondary node 220 over cross-over cable 218, if the heartbeatfails, then secondary node 220 will take over the services provided byprimary node 210 before the failure. As will be further described,however, according to an advantage of the present invention, middlewarecomponents may further analyze the heartbeat failure and provideadditional information about the failure before secondary node 220 takesover for the services provided by primary node 210. Further, as will befurther described, both Linux based and non-Linux based heartbeats maybe monitored via cross-over cable 218.

Primary node 210 and secondary node 220 access data storage systems 214and 224. Advantageously, a data replicator, herein depicted as a drbdpartition 230, including a partition of each of data storage systems 214and 224, for replicating data accessible by primary node 210 andsecondary node 220 without requiring a storage device that is actuallyphysically shared between primary node 210 and secondary node 220According to an advantage of the present invention, drbd is configuredto run on the partition to facilitate the transfer of data duringfailover from primary node 210 to secondary node 220. It will beunderstood that while the invention is described with respect to a drbdpartition managed by drbd scripts, other distributed data replicationsystems may be implemented.

Uninterrupted power supply (UPS) 212 and UPS 222 each provide anindependent power supply to primary node 210 and secondary node 220,respectively. Preferably, a connection is also established between UPS212 and secondary node 220 and UPS 222 and primary node 210. In oneembodiment, a serial cable 216 is provided from primary node 210 to UPS222 and a serial cable 226 is provided from secondary node 220 to UPS212. It will be understood, however, that other types of connectionhardware may be implemented.

According to an advantage of the present invention, when a failure isdetected in primary node 210, secondary node 220 begins receiving therequests previously directed to primary node 210 after failover. Becauseonly a portion of the hardware, software, or network running on primarynode 210 may fail, the only way to ensure that primary node 210 does nottry to update data after the failover is to turn off UPS 212.Advantageously, as will be further described, when the failover tostandby node 220 is detected, STONITH, described in more detail herein,is implemented by the cluster manager to direct a command from standbynode 220 to UPS 212 to turn off the power supply.

With reference now to FIG. 3, there is depicted a block diagram of acluster manager in accordance with the method, system, and program ofthe present invention. As illustrated, a cluster manager 322 includesmultiple components utilized to implement an efficient failoverincluding a heartbeat tool 402, drbd scripts 404, mon 406, and a stonithfunction 408. It will be understood that other components may beincluded in a cluster manager to manage other aspects of the cluster.Further, it will be understood that additional components may beincluded in cluster manager 322 to manage failover.

Heartbeat tool 402 preferably includes the heartbeat package for Linux,configured for managing failover within a HA cluster with a J2EEcompliant middleware stack. In particular, Heartbeat tool 402 generallyworks by sending a “heartbeat” request between two nodes in a cluster.As described in FIG. 2, the heartbeat request may be sent throughcross-over cable between network adapters at each node. When applied toa J2EE compliant middleware stack running on clusters of server systems,heartbeat requests sent by heartbeat tool 402 are distributed about thedifferent layers of the stack.

If the heartbeat request fails to be returned, then the secondary nodecan assume that the primary node failed and take over IP, data, andservices that were running on the primary node. When the secondary nodetakes over the IP, data, and services that were running on the primarynode, heartbeat tool 402 startups components of the secondary node thatare waiting in standby mode, assigns IP addresses to components of thesecondary node, and performs other failover tasks.

Drbd 404 is a kernel module with associated scripts that that managedata in a HA cluster for improved switching of data during failover.This is performed by mirroring a block device managed by drbd 404. Drbdis a script that loads the drbd module and configures with the IPaddresses of the relevant systems in the HA cluster and the sharedstorage device.

When applied to a J2EE compliant middleware stack, the drbd managedblock device provides storage on which the middleware stack can run.Initially, the cluster is configured and the drbd partition is mountedso that only the primary node can read or write from the drbd managedblock device. When a failover occurs, the datadisk script of drbd 404 isrun by heartbeat tool 402 to mount the drbd partition so that only thesecondary node can read/write from the drbd managed block device.

Mon 406 is a service monitoring daemon that periodically runs monitoringscripts that monitor critical system services within the J2EE compliantmiddleware stack. If a service is found to have failed or terminatedabnormally, mon 406 restarts the service to ensure that all componentsof the middleware stack remain running within the primary service.Abnormal termination may occur, for example, from programming errors orcatastrophic operating system events such as temporary critical resourceconstraints with RAM. In particular, when mon restarts a service, itrestarts a new instance of the service with a process identifier (PID)different from the dead service, but the same virtual IP address.

Stonith 406 is a function called by heartbeat tool 402 to ensure dataintegrity during failover. In particular, stonith 406 includes theconfiguration of the serial cables to UPS 212 and 222, as depicted inFIG. 2. When heartbeat tool 402 calls stonith 406, the call designatesthe node to be shutdown. Stonith sends a signal to turn off the power ofthe requested UPS.

Monitoring and configuration controller 410 includes multiple monitoringcontrollers which are specified for monitoring the status of hardwareand software within the HA clusters. According to an advantage of theinvention, status information about the multiple hardware and softwarecomponents of HA clusters is forwarded to a remote centralizedenterprise console. Preferably, monitoring and configuration controller410 supplements the Java™ Management Extensions (JMX) to monitor thehardware and software components of the HA clusters, to detectbottlenecks and potential problems, and to automatically recover thecluster from critical situations. In one embodiment, the monitoringcontrollers are enabled by Tivoli™ Monitoring which forwards monitoredinformation to a Tivoli™ Enterprise Console (TEC).

In particular, while heartbeat tool 402 and mon 406 monitor the statusof specific components and specific instances of services within thenodes, monitoring and configuration controller 410 detects theconditions monitored by these tools and detects the overall status ofthe system when heartbeat tool 402 is triggered to initiate failover ormon 406 is triggered to restart a server. Thus, monitoring andconfiguration controller 410 supplements the open source tools bycompiling the status of multiple components of the nodes when failures,errors, and non-ideal conditions occur.

According to one advantage of the invention, the remote centralizedmonitoring console can use the information gathered to determineconfiguration changes. In particular, according to an advantage of theinvention, the monitoring controllers of monitoring and configurationcontroller 410 are each configured to monitor each hardware component inthe HA cluster and each of the layers of the J2EE compliant middlewarestack. Thus, based on monitored information about the hardware andmiddleware layers, the console can determine which middleware layersneed more memory for caching requests, need more threads for handlingrequests, or need to be reconfigured in some other manner. The consolecan send configuration changes to the configuration controllers ofmonitoring and configuration controller 410, which then adjust theconfiguration of the HA clusters. In one embodiment, the configurationcontroller is a Tivoli™ Configuration Manager which manages theconfiguration characteristics of the HA clusters.

According to another advantage of the invention, in an enterprisesystem, the console use the information gathered to determine which HAclusters need hardware and software upgrades. For example, for themonitored information, the console can determine which stores havehardware which seems to be failing and needs to be replaced, whichstores have hardware which has reached capacity and needs to beupgraded, and which stores have software that is failing or not runningreliably.

According to yet another advantage of the invention, monitoring andconfiguration controller 410 interacts with the other monitoringcomponents within cluster manager 322 to gather the status informationthat is sent to the console. For example, when mon 406 detects a failureof any of the monitored services, monitoring and configurationcontroller 410 sends a notification to the remote centralized monitoringconsole so that a bigger picture of failures in the system can becompiled. Further, when heartbeat tool 402 initiates a failover of onenode of the system to another node, monitoring and configurationcontroller 410 sends a notification to the remote centralized monitoringconsole so that node failure statistics can be gathered.

With reference now to FIG. 4, there is depicted a block diagram of oneembodiment of a software configuration of a HA cluster before failoverin accordance with the method, system, and program of the presentinvention. As depicted, primary node 210 and secondary node 220represent clusters of server systems, each assigned to an IP address.

According to an advantage of the present invention, cluster manager 322runs on primary node 210 and secondary node 220 to monitor for failures,restart services, and control failover when a failure is detected. Asillustrated, cluster manager 322 sets up drbd partition 230 that islocated on storage shared between primary node 210 and secondary node220.

Primary node 210 includes all active components of the middleware stack:a load balancer 312, HTTP servers 314, web application servers (WAS)316, messaging controllers 318, and a database server 320. Secondarynode 220 includes active HTTP servers 334 and WASs 336, however, loadbalancer 332, messaging controllers 338, and database 340 are in standbymode.

Load balancers 312 and 332 preferably balance the load of requestsbetween HTTP and WAS servers, which may also be clustered. Preferably,load balancers 312 and 314 perform intelligent load balancing by usingserver availability, capability, workload, and other criteria. Accordingto one embodiment, load balancers 312 and 332 may be implemented throughthe IBM WebSphere™ Edge Server.

As illustrated, load balancers 312 and 332 may implement a heartbeatindependent of the Linux based heartbeat. Alternatively, the Linux basedheartbeat monitoring 332 and 342 may monitor the status of loadbalancers 312 and 332.

HTTP servers 314 and 334 may include clusters of servers designed toreceive HTTP requests and distribute HTTP requests among WAS 316 and336, respectively. In addition, HTTP servers 314 and 334 are enabled tocall enablers, such as servlet containers and Enterprise Java™ Bean(EJB) containers, when other requests, such as requests for servlets andEJBs, are received. According to one embodiment, HTTP servers 314 and334 may be implemented through an HTTP server bundled with IBM'sWebSphere™, and in particular WebSphere™ v. 5.0. WebSphere™ 5.0 isadvantageous because multiple copies of the WebSphere™ components can becontrolled from one location. Thus, configuration changes can be made inone place that effects multiple instances of the software componentslocated on multiple server systems.

According to an advantage of the present invention, HTTP servers 314 and334 are run in an active/active configuration where the heartbeat toolof cluster manager 322 actives HTTP server after primary node is up andrunning. By running HTTP servers 314 and 334 in an active/activeconfiguration, the request load can be split across the two (or more)servers to increase the speed at which client requests are handled. Inaddition, by running HTTP servers 314 and 334 in an active/activeconfiguration, then startup time on failover is reduced.

WAS 316 and 336 preferably include clusters of servers enabled tosupport web applications providing mission-critical services tocustomers, and in particular these servers are enabled to support J2EEapplications. According to one embodiment, WAS 316 and 336 areWebSphere™ Application Servers supported by IBM's Websphere™ 5.0 thathost the servlets, EJBs, and other J2EE components necessary forsupporting a J2EE application and services.

WAS 316 interacts with messaging controller 318 and database server 320to provide application server functionality integrated with messagingcontrol and databases. According to an advantage of the presentinvention, WAS 316 and WAS 336 are run in an active/activeconfigurationIn particular, when initializing the systems, oncemessaging controller 318 and database server 320 are available, theheartbeat tool of cluster manager 322 launches WAS 336 to create theactive/active configuration. By running an active-active configuration,the request load can be split across multiple clusters of systems toincrease the speed at which client requests are handled. In addition, byrunning an active/active configuration, then startup time on failover isreduced.

Messaging controllers 318 and 338 include a controller for listening forasynchronous requests and storing those requests in a local queue toprovide a queue to communicate with J2EE based systems. Messagingcontroller 318 and 338 may implement IBM MQSeries™, IBM WebSphere™ MQ,or other message controllers that supplement the Java™ Messaging Service(JMS).

According to an advantage of the present invention, messagingcontrollers 318 and 338 are run in an active/standby configuration wherethe drbd of cluster manager 322 manages the persistent resources in themessaging queue in drbd partition 230 and the heartbeat tool of clustermanager 322 controls the startup of messaging controller 338 in afailover.

Database servers 320 and 340 provide control for persistent storage.Database servers 320 and 340 may be implemented through a databasecontrol system such as IBM DB2 UDB Enterprise Edition or otherrelational database management systems.

According to an advantage of the present invention, database servers 320and 340 are run in an active/standby configuration where the drbd ofcluster manager 322 manages the persistent resources in the database indrbd partition 230 and the heartbeat tool of cluster manager 322controls the startup of database server 340 in a failover.

For messaging controllers 318 and 338 and database servers 320 and 340to run in active/standby configuration and quickly failover with minimaldata loss, messaging controller 318 and database server 320 areconfigured to point to the location where drbd partition 320 is mountedas the root for storage of the queue and database. In addition, clustermanager 322 configures drbd and the heartbeat tool with the virtual IPaddress of messaging controller 318 and database server 320.

Further, according to an advantage of the present invention, the monfunction of cluster manager 322 periodically runs monitoring scriptsthat monitor critical system services, such as the services provided bymessaging controller 318 and database server 320. If a service is foundto have failed or terminated abnormally, mon restarts the service toensure that all components of the middleware stack remain running withinthe primary service.

It is important to note that the method of configuring each level ofmiddleware to achieve efficient failover and controlling each level ofmiddleware through cluster manager 322 may be applied to other types ofmiddleware. Thus, as the functions available from a middleware softwarestack that is J2EE compatible continue to expand, each middlewarecomponent can be configured either in an active/active or active/standbyconfiguration, monitored by cluster manager 322, and controlled duringfailover.

With reference now to FIG. 5, there is depicted a block diagram of oneembodiment of a software configuration of a HA cluster after failover inaccordance with the method, system, and program of the presentinvention. As depicted, after failover, primary node 210 is marked as afailed node. Secondary node 220 takes over as the all active node.

When a failure is detected and secondary node 220 designates primarynode 210 as “dead”, hardware and software issues are present. Inparticular, primary node 210 may not respond to a heartbeat requestwithin the required time, however, would be operational shortlythereafter. To avoid a case where primary node 210 and secondary node220 are both operational, as previously described, the heartbeat tool ofcluster manager 322 will call STONITH to turn off the UPS to primarynode 210. By implementing an inexpensive UPS, controllable by STONITH,data integrity can be achieved and the “split brain” issue of HA thatcan arise when the primary node is not really dead is avoided.

Next, during failover, the load balancer heartbeat manages startup ofload balancer 332. When activated, the heartbeat tool of cluster manager322 assigns the virtual IP1 address of primary node 210 to load balancer332. Accordingly, requests to the virtual IP address are redirected toload balancer 332 so that no change in the IP address of the loadbalancing cluster occurs.

During failover, since HTTP server 334 and WAS 336 are already active,the heartbeat tool of cluster manager 322 does not need to startup thesecomponents. However, since messaging controller 338 and database server340 are in standby, the heartbeat tool of cluster manager 322 needs tomanage the failover of these layers. First, the heartbeat tool willtakeover the virtual IP2 address. Next, the heartbeat tool will startthe datadisk service of the drbd to configure and mount the drbdmirrored partition. Finally, the heartbeat tool will startup messagingcontroller 338 and database server 340 configured to virtual IP2 addressand with the message queue and database instances launching on mirroreddrbd partition 230. Alternatively, although not depicted, databaseserver 340 may be in active mode, rather than standby, because thevirtual IP2 address is only available to one node at a time. Sincedatabase server 340 does not attempt to touch the data on drbd partition230 until a request arrives, at failover, database server 340 isconfigured to the virtual IP2 address and mirrored drbd partition 230 isaccessible before a request arrives. In contrast, some layers, such asmessaging controller 338 load data directly at startup and thus willcrash if started up on secondary node 220 before failover because dataon drbd partition 230 is not available to secondary node 220 beforefailover.

Referring now to FIG. 6, there is depicted a block diagram of oneexample of an implementation of an independent software vendor (ISV)application within a J2EE compliant middleware in a HA system. Asillustrated, an active WAS 602, active IBM MQSeries™ server 610, andactive IBM DB2 server 614 illustrate a portion of the primary node of aJ2EE compliant middleware stack interfacing with a drbd partition 630.As illustrated at reference numeral 620, an item sale or transactioncompletion is received at an active WebSphere™ Application Server 602.An ISV may program a servlet or EJB to handle a particular type ofincoming request. For example, as depicted at reference numeral 620, alookup servlet 604 is an ISV web application that handles price lookups(PLU) to check the price of items as they are scanned at a cashregister. Lookup servlet 602 then posts a request for the retaintransaction to be completed asynchronously by another component, such astransaction servlet 608 or another servlet or EJB. First, however, asdepicted at reference numeral 622, the information is transferred to MQlistener 612 and placed on MQ queue 632 to free lookup servlet 604 toreceive the next incoming request and to ensure that the transactionwill be recorded exactly once, in order, via MQ queue 632. Next, asdepicted at reference numeral 624, MDB 606 is then called to take thetransaction off MQ queue 632 and as depicted at reference numeral 626,to feed the transaction to transaction servlet 626. Transaction servlet626 ultimately processes the PLU and, as depicted at reference numeral628, commits the result to IBM DB2 controller 616 for storage in DB2634.

In particular, FIG. 6 illustrates the advantages of the J2EE compliantmiddleware stack in a HA system during failover because the stackensures that each transaction will be recorded exactly once, even iffailover occurs after a request has already begun to transition betweenthe layers of the stack. In addition, FIG. 6 illustrates the advantagesof the J2EE compliant middleware stack in a HA system during failoverbecause active layers MQSeries™ server 610 and DB2 server 614 interfacewith drbd partition 630 that is only accessible to the primary node, butis quickly remounted for access by the secondary node during failover.

With reference now to FIG. 7, there is depicted a high level logicflowchart of a process and program for configuring a drbd partition to aJ2EE compliant middleware stack in a HA cluster. As depicted, theprocess starts at block 700 and thereafter proceeds to block 702. Block702 depicts configuring and mounting the drbd partition. Next, block 704depicts activating the message queue and database on the drbd partition.Thereafter, block 706 illustrates recording the virtual IP address ofthe messaging server and database server accessing the drbd partitionfor efficient transfer of access to the drbd partition during failover,and the process ends.

Referring now to FIG. 8, there is depicted a high level logic flowchartof a process and program for controlling configuration and failover of aJ2EE compliant middleware stack in a HA cluster through a heartbeatcontroller. As depicted, the process starts at block 800 and thereafterproceeds to block 802. Block 802 illustrates activating the middlewarelayers of the primary node. Thereafter, block 804 depicts activating theHTTP server and the WAS middleware layers of the secondary node. Inaddition, other middleware layers that are designated to run in anactive-active configuration are activated. Thereafter, block 806 depictsperiodically initiating a heartbeat request from the secondary node tothe primary node. Block 808 depicts a determination whether a heartbeatreturn is detected by the secondary node. If a heartbeat return isdetected, then the process returns to block 806. If a heartbeat returnis not detected, then the process passes to block 810.

Block 810 depicts calling STONITH to turn off the power supply of theprimary node. Next, block 812 depicts taking over the virtual IPaddresses from the primary node to assign to the redundant component inthe secondary node. Thereafter, block 814 depicts calling the datadiskscript to remount the drbd partition for access by the secondary nodeand the process ends. Then, block 816 depicts activating the standbymiddleware layers on the secondary node and launch data on the drbdpartition. It will be understood that additional steps may be performedby the heartbeat tool and other cluster management services duringfailover.

With reference now to FIG. 9, there is depicted a high level logicflowchart of a process and program for controlling a mon function formonitoring services provided by a J2EE compliant middleware stack. Asdepicted, the process starts at block 900 and thereafter proceeds toblock 902. Block 902 depicts configuring a schedule for monitoringservices provided by the middleware. Next, block 904 depicts adetermination whether a scheduled monitoring time is triggered. If ascheduled monitoring time is not triggered, then the process iterates atblock 904. If a scheduled monitoring time is triggered, then the processpasses to block 906. Block 906 depicts monitoring the status of thescheduled service. Thereafter, block 908 depicts a determination whetherthe service is detected as dead or failed in some manner. If the serviceis not detected as dead, then the process ends. If the service isdetected as dead, then the process passes to block 910. Block 910depicts restarting the same service with a new PID, and the processends.

Referring now to FIG. 10, there is depicted a block diagram of anenterprise network including multiple HA systems running J2EE middlewarestacks and managed by a remote enterprise console in accordance with themethod, system, and program of the present invention. As depicted, a HAsystem 1202 and a HA system 1204 are communicative connected to a remoteenterprise console 1210 that monitors and remotely controls HA systems1202 and 1204 via network 102. It will be understood that multiple HAsystems may be monitored and controlled by a single or multiple remotecentral consoles.

According to an advantage of the invention, each of HA systems 1202 and1204 may handle retail transactions and other mission criticaloperations. According to one embodiment, each of HA systems 1202 and1204 enable high availability through redundant J2EE compliantmiddleware stacks that enable J2EE applications, such as the middlewarestacks illustrated in FIGS. 4 and 5. In particular, each of HA systems1202 and 1204 includes a cluster manager running monitoring andconfiguration controllers 410, as depicted in FIG. 3.

Advantageously, when errors, failures, or non-ideal conditions occur atany of HA systems 1202 and 1204, monitoring and configurationcontrollers 410 detects the condition of the system at the time of theerror, failure or other non-ideal condition and then compiles theinformation to make a report to remote enterprise console 1210.According to an advantage of the invention, if the heartbeat monitor ormon functions detect a failure or error, then monitoring andconfiguration controllers 410 are triggered to detect the failure orerror and determine the system conditions at the time of the failure orerror.

Remote enterprise console 1210 preferably stores monitored informationin a database. Next, remote enterprise console 1210 preferably includesa first controller for analyzing the error and failure informationreceived from HA systems 1202 and 1204 and potentially returningconfiguration changes to the HA systems to attempt to prevent andimprove the efficiency of failovers. In addition, remote enterpriseconsole 1210 may include a second controller that compares the failures,errors and other information received from multiple HA systems todetermine which systems need repairs and upgrades and which systems arenot meeting performance requirements. Remote enterprise console 1210 maygather and control display of performance statistics for HA systems 1202and 1204.

Referring now to FIG. 11, there is depicted a high level logic flowchartof a process and program for controlling a monitoring controller withina HA cluster manager in accordance with the method, system, and programof the present invention. As depicted, the process starts at block 1000and thereafter proceeds to block 1002. Block 1002 depicts adetermination whether a failure or error is detected from the heartbeatmonitor, mon, or other monitoring controller monitoring a middlewarestack in a HA system. If no failure or error is detected, then theprocess iterates at block 1002. If a failure or error is detected, thenthe process passes to block 1004. Block 1004 depicts gathering andanalyzing available system information at the time of the failure orerror. Next, block 1006 depicts sending the failure or error andavailable system information to a remote central console monitoring theHA system, and the process ends.

With reference now to FIG. 12, there is depicted a high level logicflowchart of a process and program for remotely controlling a clustermanager of an HA system to reconfigure the HA system. As illustrated,the process starts at block 1100 and thereafter proceeds to block 1102.Block 1102 depicts a determination of whether a configuration request isreceived from a remote enterprise console to reconfigure the HA systemrunning a middleware stack. If the request is not received, then theprocess iterates at block 1102. If the request is received, then theprocess passes to block 1104. Block 1104 depicts calling the heartbeatmonitor to reconfigure the HA system failover settings, and the processends. In addition, other controllers within the cluster manager of a HAsystem may be called to adjust other software and hardwareconfigurations of the HA system.

Referring now to FIG. 13, there is depicted a high level logic flowchartof a process and program for controlling a remote enterprise console formanaging multiple HA systems in a cluster. As depicted, the processstarts at block 1300 and thereafter proceeds to block 1302. Block 1302depicts a determination whether monitored information is received froman HA system. If monitored information is not received, then the processiterates at block 1302. If monitored information is received, then theprocess passes to block 1304. In particular, the remote enterpriseconsole may periodically send request to each of the HA system formonitored information and each HA system may also automatically sendmonitored information.

Block 1304 depicts adding the monitored information to an enterprisedatabase in which monitored information from multiple HA systems isstored. Next, block 1306 depicts requesting reconfiguration of the HAsystem if the monitored information triggers reconfiguration. Inparticular, a remote enterprise console may include predeterminedconfigurations to be requested when particular types of errors aredetected in monitored information. Alternatively, a system administratormay recommend the type of configuration for a particular type of error.Thereafter, block 1308 depicts recalculating the performance statisticsfor the HA system based on the monitored information. In particular,calculation of performance statistics may only be triggered for certaintypes of monitored errors or fluctuations. Next, block 1312 depictscomparing the performance of this HA system with the performance of theother HA systems in the enterprise network and the performancerequirements set for the enterprise network. Then, block 1314 depictsdisplaying the comparative performance results in charts and graphs. Forexample, a chart may depict graphical representations of the locationsof the HA systems and provide a graphical indicator of which systemshave failed and provide graphical indicators to show the performance ofeach HA system relative to the other HA systems. Further, real-timeperformance of each system and any errors reported may be displayed.Next, block 1316 depicts recommending corrective action for HA systemweaknesses, and the process ends. For example, the recommendations mayindicate which HA systems need to be replaced, which HA systems need tobe upgraded, and which HA systems need software upgrades or fine tuning.It will be understood that the processes depicted in FIG. 13 areexamples of types of processes that can be performed on the monitoredinformation received from multiple high availability servers and thatother similar analysis and outputs can be performed without departingfrom the scope of the invention.

While the invention has been particularly shown and described withreference to a preferred embodiment, it will be understood by thoseskilled in the art that various changes in form and detail may be madetherein without departing from the spirit and scope of the invention.

1. A system for enabling remote enterprise management of highavailability systems, comprising: a particular high availability systemof a plurality high availability systems communicatively connected to aremote enterprise server via a network; said particular highavailability system further comprising: a primary node running amiddleware stack for supporting web applications, wherein a plurality oflayers of said middleware stack are active, wherein said primary node isassigned a virtual IP address to which requests are directed; asecondary node running a redundant middleware stack for mirroring saidplurality of layers of said middleware stack of said primary node,wherein a first selection of said plurality of layers of said redundantmiddleware stack are active and a second selection of said plurality oflayers of said redundant middleware stack are in standby; a datareplication partition shared between said primary node and saidsecondary node with data accessible to a selection of said plurality oflayers of said active middleware stack, wherein said selection of saidplurality of layers of said active middleware stack correspond to saidsecond selection of said plurality of layers of said redundantmiddleware stack in standby; a cluster management controller formonitoring a status of said primary node of said high availabilitysystem and responsive to said status indicating an error, transferringsaid virtual IP address from said primary node to said secondary node,turning off power to said primary node, remounting said data replicationpartition for access by said secondary node, and activating said secondselection of said plurality of layers of said redundant middleware stackrequiring access to said data within said data replication partition;and a monitoring controller for detecting when said cluster managementcontroller reacts to said error in said primary node and detecting acondition of a plurality of components of said high availability systemat a time of said error, wherein said monitoring controller reports saiderror and said condition of said plurality of components to said remoteenterprise server enabled to manage said plurality of high availabilitysystems based on a separate report received from each of said pluralityof high availability systems.
 2. The system according to claim 1 forenabling remote enterprise management of high availability systems, saidparticular high availability system further comprising: said primarynode and said secondary node each comprising a plurality of serversimplementing a J2EE compliant middleware stack monitored by said clustermanagement controller.
 3. The system according to claim 1 for enablingremote enterprise management of high availability systems, said clustermanagement controller further comprising: a heartbeat monitor fordetecting a status of said primary node of said particular highavailability system.
 4. The system according to claim 1 for enablingremote enterprise management of high availability systems, said clustermanagement controller further comprising: a service monitor daemon fordetecting a status of a service provided by said middleware layer ofsaid primary node of said particular high availability system.
 5. Thesystem according to claim 1 for enabling remote enterprise management ofhigh availability systems, wherein said monitoring controller receives aconfiguration request from said remote enterprise server and adjusts aconfiguration for how said cluster management controller will react toadjust said particular high availability system in response to a futureerror.
 6. The system according to claim 1 for enabling remote enterprisemanagement of high availability systems, wherein said monitoringcontroller receives a configuration request from said remote enterpriseserver and adjusts a hardware configuration of said particular highavailability system according to said request.
 7. A method for enablingremote enterprise management of high availability systems, comprising:monitoring a status of at least one component of a particular highavailability system from among a plurality of high availability systems,communicatively connected to a remote enterprise server via a network;running, at a primary node within said particular high availabilitysystem, a middleware stack for supporting web applications, wherein aplurality of layers of said middleware stack are active, wherein saidprimary node is assigned a virtual IP address to which requests aredirected; running, at a secondary node within said particular highavailability system, a redundant middleware stack for mirroring saidplurality of layers of said middleware stack of said primary node,wherein a first selection of said plurality of layers of said redundantmiddleware stack are active and a second selection of said plurality oflayers of said redundant middleware stack are in standby; sharing,between said primary node and said secondary node, a data replicationpartition with data accessible to a selection of said plurality oflayers of said active middleware stack, wherein said selection of saidplurality of layers of said active middleware stack correspond to saidsecond selection of said plurality of layers of said redundantmiddleware stack in standby; responsive to said status indicating anerror, reacting to adjust said particular high availability system bytransferring said virtual IP address from said primary node to saidsecondary node, turning off power to said primary node, remounting saiddata replication partition for access by said secondary node, andactivating said second selection of said plurality of layers of saidredundant middleware stack requiring access to said data within saiddata replication partition; detecting a condition of a plurality ofcomponents of said high availability system at a time of said error; andreporting said error and said condition of said plurality of componentsto said remote enterprise server enabled to manage said plurality ofhigh availability systems based on a separate report received from eachof said plurality of high availability systems.
 8. The method accordingto claim 7 for enabling remote enterprise management of highavailability systems, further comprising: monitoring said status of saidcomponent by monitoring said primary node and said secondary node eachcomprising a plurality of servers implementing a J2EE compliantmiddleware stack.
 9. The method according to claim 7 for enabling remoteenterprise management of high availability systems, further comprising:monitoring, by a heartbeat monitor, the status of said primary node ofsaid high availability system.
 10. The method according to claim 7 forenabling remote enterprise management of high availability systems,further comprising: detecting, by a service monitor daemon, a status ofa service provided by said middleware layer of said primary node of saidparticular high availability system.
 11. The method according to claim 7for enabling remote enterprise management of high availability systems,further comprising: receiving a configuration request from said remoteenterprise server; and adjusting a configuration for how said clustermanagement controller will react to adjust said particular highavailability system in response to a future error.
 12. The methodaccording to claim 7 for enabling remote enterprise management of highavailability systems, further comprising: receiving a configurationrequest from said remote enterprise server; and adjusting a hardwareconfiguration of said particular high availability system according tosaid request.
 13. A computer program product, residing on a computerreadable medium, for enabling remote enterprise management of highavailability systems, comprising: means for monitoring a status of atleast one component of a particular high availability system from amonga plurality of high availability systems communicatively connected to aremote enterprise server via a network; means for running, at a primarynode within said particular high availability system, a middleware stackfor supporting web applications, wherein a plurality of layers of saidmiddleware stack are active, wherein said primary node is assigned avirtual IP address to which requests are directed; means for running, ata secondary node within said particular high availability system, aredundant middleware stack for mirroring said plurality of layers ofsaid middleware stack of said primary node, wherein a first selection ofsaid plurality of layers of said redundant middleware stack are activeand a second selection of said plurality of layers of said redundantmiddleware stack are in standby; and means for sharing, between saidprimary node and said secondary node, a data replication partition withdata accessible to a selection of said plurality of layers of saidactive middleware stack, wherein said selection of said plurality oflayers of said active middleware stack correspond to said secondselection of said plurality of layers of said redundant middleware stackin standby; means, responsive to said status indicating an error, forreacting to adjust said particular high availability system bytransferring said virtual IP address from said primary node to saidsecondary node, turning off power to said primary node, remounting saiddata replication partition for access by said secondary node, andactivating said second selection of said plurality of layers of saidredundant middleware stack requiring access to said data within saiddata replication partition; and; means for detecting a condition of aplurality of components of said high availability system at said time ofsaid error; and means for reporting said error and said condition ofsaid plurality of components to said remote enterprise server enabled tomanage said plurality of high availability systems based on a separatereport received from each of said plurality of high availabilitysystems.
 14. The computer program product according to claim 13 forenabling remote enterprise management of high availability systems,further comprising: means for monitoring said status of said componentby monitoring said primary node and said secondary node each comprisinga plurality of servers implementing a J2EE compliant middleware stack.15. The computer program product according to claim 13 for enablingremote enterprise management of high availability systems, furthercomprising: means for monitoring, by a heartbeat monitor, the status ofsaid primary node of said high availability system.
 16. The computerprogram product according to claim 13 for enabling remote enterprisemanagement of high availability systems, further comprising: means fordetecting, by a service monitor daemon, a status of a service providedby said middleware layer of said primary node of said particular highavailability system.
 17. The computer program product according to claim13 for enabling remote enterprise management of high availabilitysystems, further comprising: means for receiving a configuration requestfrom said remote enterprise server; and means for adjusting aconfiguration for how said cluster management controller will react toadjust said particular high availability system in response to a futureerror.
 18. The computer program product according to claim 13 forenabling remote enterprise management of high availability systems,further comprising: means for receiving a configuration request fromsaid remote enterprise server; and means for adjusting a hardwareconfiguration of said particular high availability system according tosaid request.
 19. A system for remotely configuring a plurality of highavailability systems, comprising: a plurality of high availabilitysystems communicatively connected to a network, each of said pluralityof high availability systems further separately comprising: a primarynode running a middleware stack for supporting web applications, whereina plurality of layers of said middleware stack are active, wherein saidprimary node is assigned a virtual IP address to which requests aredirected; a secondary node running a redundant middleware stack formirroring said plurality of layers of said middleware stack of saidprimary node, wherein a first selection of said plurality of layers ofsaid redundant middleware stack are active and a second selection ofsaid plurality of layers of said redundant middleware stack are instandby; a data replication partition shared between said primary nodeand said secondary node with data accessible to a selection of saidplurality of layers of said active middleware stack, wherein saidselection of said plurality of layers of said active middleware stackcorrespond to said second selection of said plurality of layers of saidredundant middleware stack in standby; a cluster management controllerfor monitoring a status of said primary node and responsive to saidstatus indicating an error, transferring said virtual IP address fromsaid primary node to said secondary node, turning off power to saidprimary node, remounting said data replication partition for access bysaid secondary node, and activating said second selection of saidplurality of layers of said redundant middleware stack requiring accessto said data within said data replication partition; a monitoringcontroller for detecting monitored information about a plurality ofcomponents of each separate one of said plurality of high availabilitysystems at a same time as said status indicates an error; and a remoteenterprise server communicatively connected to said network, whereinsaid remote enterprise server receives said monitored information abouteach of said plurality of high availability systems from each saidseparate monitoring controller, analyzes said monitored information, andsends requests for reconfiguration to each of said plurality of highavailability systems which submit monitored information indicatingerrors which can be adjusted by reconfiguration.